Field
The present disclosure relates generally to a method, apparatus, system, and computer readable media for optimizing storage of information in both on-disk and in-memory representation, and more particularly, relates to optimized, sequential organization in files for both on-disk and in-memory representations of information
Background
Traditional datastores and databases use sequential log files and paged datastore/database files. This approach has many weaknesses that are difficult if not impossible to overcome without significant architectural and algorithmic changes. Such drawbacks include severe performance degradation with random access patterns; seeks occurring to random pages even with sequential data; data being written at least twice, once to the log file(s) and again to the datastore/database file(s); system startup and shutdown being very slow as log files are read/purged and error detection and correction is performed; and error recovery being very complex since data can be partially written to existing pages.
Additionally, traditional designs store operations and data in log files. Thereafter, this information is moved to paged database files by reprocessing the operations and data and/or purging memory. Pages are overwritten in the process.